Apparatus and method for creating effects in video

ABSTRACT

Creating a blur in a digital image by receiving original pixel values in a frame of image data, and for each pixel position in the digital image being blurred, calculating a blurred pixel value by applying a box filter having a plurality of elements with unity multiplier values to original pixel values for pixel positions that include the pixel position and pixel positions that are nearby to it. The output of the first box filter can be sequentially applied as inputs to a second box filter.

TECHNICAL FIELD

[0001] The invention relates to modifying video.

BACKGROUND

[0002] When a video program is created at a nonlinear video editing andcompositing system, analog video source materials are typicallydigitized, compressed and stored on disc for later random access for usein creating a video program. Digital source video can also be input intothe nonlinear video editor. Segments of the video, also referred to as“video clips,” can be taken from different sources, and the beginningand end points can be trimmed as desired during editing. The order ofclips is established in a time line to determine the sequence, and theposition of a clip can be changed. Clips can be deleted or inserted asdesired. During the editing process, the clips of compressed video canbe randomly accessed, decompressed and viewed in real time. The sourcevideo can be modified by various effects such as scale, rotate, blurusing hardware or software filters to manipulate pixels on the fly inreal time or offline.

[0003] A blur can be created by replacing the value for a pixel with anew value based upon the original values for that pixel and nearbypixels, with the values of the closer pixels being weighted more thanthe value of pixels that are further away. The new pixel value P_(blur)is a convolution, i.e., a sum of weighted values, given by the followingformula:

P_(blur)=ΣP_(i)*W_(i)

[0004] Where: P_(i) equals the value of pixel i, which is a neighbor ofthe pixel being calculated or the pixel itself, and

[0005] W_(i) equals the weight for pixel i.

[0006] If the weights are calculated using a Gaussian distribution, ablur will simulate a defocused image and be visually pleasing. TheGaussian weights Wgi can be calculated using the following formula:

W_(gi)=e^(−x**2/c**2),

[0007] Where: x is the distance from pixel i to the pixel beingcalculated, and c is a constant.

[0008] Calculation of each blurred pixel value requires n multiplyoperations and n add operations, where n is the number of pixels used inthe blur. E.g., if the blur were based on the values of pixels in 101pixels in the same row and 101 pixels in the same column, there would be202 multiplications and 202 additions to obtain a single blurred pixelvalue. This process would then need to be repeated for each pixel beingblurred in each frame of the video program being subjected to the blur.

SUMMARY

[0009] In one aspect, the invention features, in general, creating ablur in a digital image by receiving original pixel values in a frame ofimage data, and for each pixel position in the digital image beingblurred, calculating a blurred pixel value by applying a box filterhaving a plurality of elements with unity multiplier values to originalpixel values for pixel positions that include the pixel position andpixel positions that are nearby to it.

[0010] In another aspect, the invention features, in general, creating ablur in a digital image by calculating, for each pixel position in thedigital image being blurred, a blurred pixel value by sequentiallyapplying the original pixel values as inputs to a first box filter thathas a plurality of elements and by sequentially applying the outputs ofthe first box filter as the inputs to a second box filter.

[0011] Particular embodiments of the invention may include one or moreof the following features. The outputs of the second box filter can besequentially applied as the inputs to a third box filter. Additional boxfilters can be added if desired. The elements of the box filters canhave unity multiplier values. Each box filter can be implemented by astorage area that maintains a current filter output and an adder andsubtracter to update the filter output each time that a new input isreceived by g the new input to the current filter output and subtractingthe input that had been received m pixel positions earlier, where m isthe number of elements in the box filter.

[0012] Embodiments of the invention may have one or more of thefollowing advantages. Gaussian type blurs can be simply and quicklycreated with a small number of operations and/or a small number ofhardware components by employing multiple box filters in series.

[0013] Other aspects and features of the invention will be apparent fromthe detailed description below and from the claims.

DESCRIPTION OF DRAWINGS

[0014]FIG. 1 is a box diagram of a video editing and compositing system.

[0015]FIG. 2 is a diagram of a frame of pixels.

[0016]FIG. 3 is a diagram of a blur effects subsystem of the FIG. 1system.

[0017]FIG. 4 is a representation illustrating the operation of afive-element box filter that has unity (1.0) multiplier values.

[0018]FIG. 5 is a representation illustrating application of the FIG. 4box filter to a row of pixel data.

[0019]FIG. 6 is a diagram of components of the FIG. 4 box filter.

[0020]FIG. 7 is a representation of the five-component sequence thatresults from passing a single pixel value of 1.0 through thefive-element box filter of FIG. 4.

[0021]FIG. 8 is a representation of the nine-component sequence thatresults from passing the five-component output of FIG. 6 through afive-element box filter that has unity (1.0) multiplier values.

[0022]FIG. 9 is a representation of the seventeen-component sequencethat results from passing the nine-component output of FIG. 7 through anine-element box filter that has unity (1.0) multiplier values.

[0023]FIG. 10 is a graph illustrating the weighting functions resultingfrom use of different numbers of box filters in sequence.

[0024]FIG. 11 is an alternative embodiment of a blur effects subsystemused to create a blur effect on image data.

DETAILED DESCRIPTION

[0025] Referring to FIG. 1, nonlinear video editing and compositingsystem 10 includes host computer 12 having monitor 14, user inputdevices 16 (e.g., keyboard and mouse), video editing and compositingsoftware 18, video editing peripheral board 20 in an expansion slot ofcomputer 12, plug-in video hardware filters 22 in expansion slots ofcomputer 12 or otherwise connected thereto, video tape recorder (VTR)24, and mass storage disc 26.

[0026]FIG. 2 shows a five-by-five block 100 of pixels 102 in a frame 104of image data processed in video editing and compositing system 10.Central pixel 102H is located in column 106 and row 108. In creating ablur effect for the frame of image data, the value for each pixel beingblurred is replaced by a new value based upon that pixel's originalvalue and the value of nearby pixels in the same column and same row.System 10 employs sequential box filter operations to create modifiedpixel values with a reduced number of mathematical operations ascompared to the convolution procedure described earlier.

[0027] Referring to FIG. 3, video effects subsystem 101 of video editingand compositing system 10 includes first and second frame buffers 110,112, two five-element box filters 114, 116, a nine-element box filter118, and a divider 119. Video effects subsystem 101 can be implementedin a plug-in hardware filter 22, in other hardware, or in software 18(using random access memory in computer 12 for the frame buffers).

[0028] Referring to FIG. 4, five-element box filter 114 has fiveelements 120, each having unity (1.0) multiplier values. Filter 114receives pixel values P in sequence, and the values are passed from oneelement 120 to the next. Each element 120 provides an intermediateresult 122, which is the product of 1.0 times the pixel value presentedto that element 120 at a particular time. The output 124 of box filter114 at each time is the sum of intermediate results 122 at that time.FIG. 4 shows the pixel values P_(n−2), P_(n−1), P_(n), P_(n+1), P_(n+2)as the intermediate results for the case where the value for pixel n,P_(n), is in the central position. Output 124 for pixel n is ΣP_(n),which is given by the following formula:

ΣP _(n) =P _(n−2) +P _(n−1) +P _(n) +P _(n+1) +P _(n+2)

[0029] Referring to FIG. 5, the application of box filter 114 to aportion of row 108 of pixel values is shown. The pixel values include:

[0030] P_(n−3)P_(n−2)P_(n−1)P_(n)P_(n+1)P_(n+2)P_(n+3)P_(n+4)

[0031] Bracket 130 shows the five pixel values present in elements 120of box filter 114 (which are also the intermediate values 122, owing tothe unity multiplier values for each element) for the case when pixel nis in the central position so as to provide and output of ΣP_(n).Bracket 132 shows the five pixel values present in elements 120 of boxfilter 114 (which are also the intermediate values 122) for the casewhen pixel n+1s in the central position so as to provide an output ofΣP_(n+1). ΣP_(n) is given by the formula set forth above. ΣP_(n+1) isgiven by the following formula:

ΣP _(n+1) =P _(n−1) +P _(n) +P _(n+1) +P _(n+2) +P _(n+3)

[0032] It is seen that the filter output for pixel n+1 is related to theoutput for pixel n in that P_(n−2) has been dropped from the left andP_(n+3) is has been added at the right. ΣP_(n+1) can thus be derivedfrom ΣP_(n) using the following formula:

ΣP _(n+1) =ΣP _(n) +P _(n+3) −P _(n−2)

[0033] An adder, a subtracter, and a storage element to store thecurrent value of ΣP can thus implement box filter 114. Referring to FIG.6, box filter 114 is shown implemented by adder 134, subtracter 135, andregister 136. Temporary storage 137 (the same size as the number ofelements 122 in the box filter times the bits in each pixel value) isused to provide the values being subtracted without the need to accessthe frame buffer 110, which is a less preferred alternative. Logic 138is used to handle the situation at the beginning and end of lines beforethe elements 122 of the box filter have been filled up. Logic 138 caneither provide constants (e.g., 0 or black) or mirror the values insidethe image at the edge. After this priming to get the initial values forΣP at the beginning of a line, each new ΣP can be determined by addingthe new value at the right at adder 134 and subtracting the left-mostvalue, which, for a box filter having m elements, is the value that hadbeen received m pixel positions earlier, at subtracter 135. Box filtersof additional elements, e.g., nine-element filter 118 (or even muchlarger filters with, e.g., 101 elements) can also be implemented withthe same components. The box filter shown in FIG. 6 can be implementedin hardware or software, and result in a video effects subsystem that ismore efficient in transistors (for hardware) and time (for software)than prior approaches.

[0034] Box filter 114 can be applied to the original pixel data in therows in frame buffer 110, processing one row at time and placing themodified pixel values in the corresponding positions in frame buffer 112to blur the pixel values in a horizontal manner. Box filter 114 can thenbe applied to the modified pixel values in frame buffer 112, processingone column at a time and placing the modified pixel values in thecorresponding positions in frame buffer 110 to create a blur effect withboth horizontal and vertical blurring. The values could be divided by 5(implemented as a multiply times the inverse) before storage in eachframe buffer, or by 10 before storage in one of the frame buffers, toprovide scaling. (The values could also be scaled upon final removalfrom or prior to original storage in a frame buffer.) While such aneffect can be implemented with a small number of components and a smallnumber of operations, equal weight is given to all twenty-five originalvalues in the 5×5 box in which the pixel is centered. Such equal weightblurring is generally not as desirable as blurring that gives a higherweight to closer pixel positions.

[0035] FIGS. 7-9 illustrate how box filters can be used in series inorder to provide gradually increased weighting for closer pixelpositions. For purposes of the illustration, assume that there is asingle pixel value of 1.0 at one pixel position n in frame buffer 110,and zero values for all other pixel positions in a sequence (i.e., rowor column) of original pixel data. FIG. 7 shows the sequence thatresults from passing the single pixel value of 1.0 through thefive-element box filter 114 of FIGS. 3 and 4. It is seen that thisresults in values of 1.0 being output five times. If these output valuesare stored in a frame buffer 112, the 1.0 value of the pixel at positionn is spread out to two neighbors on each side. In implementation, thiswould result in a constant weighting, as shown by curve 200 on theweighting graph shown in FIG. 10.

[0036]FIG. 8 shows what happens when the result of FIG. 6 (values of 1.0at five adjacent positions, and zero for all other pixel positions) isthen passed through a second five-element box filter (e.g., 116) havingunity (1.0) multiplier values. The filter output 124 (UP) for the pixellocated at two pixel positions before the five pixel positions thatinclude pixel n at the center will be based upon a single 1.0 value inthe right most element 120, and thus have the value 1.0. The next pixelposition will have two elements 120 with 1.0 in each one, such that thesum is 2.0. The next pixel position will have three elements 120 with1.0 and a sum of 3.0. The next will have four and a sum of 4.0, and thenext, the central pixel n, will have five elements 120 with a 1.0, and asum of 5.0. As the pixel value sequence (1.0, 1.0, 1.0, 1.0, 1.0)continues through the box filter, the next will have four, and the nextthree and so on so that the sequence shown in FIG. 8 results. Thesefurther modified pixel values could be stored in frame buffer 110,replacing the original values at the corresponding positions. This showsweighting with pixel n getting the most weight, and the positions nextto it getting gradually less weight in a linear manner, as illustratedby curve 202 on the weighting graph shown in FIG. 10.

[0037]FIG. 9 shows what happens when the result of FIG. 8 (values of1.0, 2.0, 3.0, 4.0, 5.0. 4.0, 3.0, 2.0, 1.0 at nine adjacent positions,with n at the center, and with zero values for all other pixelpositions) is then passed through a nine-element box filter having unity(1.0) multiplier values. The filter output 124 (UP) for the pixellocated at four pixel positions before the nine pixel positions thatinclude pixel n at the center will be based upon a single 1.0 value inthe right-most element 120, and thus have the value 1.0. The next pixelposition will have one element 120 with 1.0 and another with 2.0, suchthat the sum is 3.0. The next pixel position will have three elements120 with 1.0, 2.0 and 3.0 and a sum of 6.0, and so on as shown in FIG.8. These further modified pixel values could be stored in frame buffer112, replacing the values at the corresponding positions. This showsweighting with pixel n getting the most weight, and the positions nextto it getting gradually less weight in very close to a Gaussiandistribution, as illustrated by curve 204 on the weighting graph shownin FIG. 10.

[0038] FIGS. 7-9 demonstrate how a single pixel value of 1.0 at oneposition gets distributed to seventeen pixels (eight on each side) afterpassing through two five-element box filters 114, 116 and onenine-element box filter 118, all having elements 120 with 1.0 (unity)multipliers. If an actual pixel value were used as the single inputinstead of 1.0, it would have been effectively multiplied times thenumbers shown in FIG. 9 owing to the additions that result in thesequential passage of inputs through the elements of the box filters. Ifactual non-zero pixel values were located at all pixel positions in therow (or column), they would similarly be distributed over nearby pixelswith the same weighting. Thus the values at P_(n−1) and P_(n+1) wouldeffectively be given weights of 24 and added to 25 times the value atP_(n), and the values at P_(n−2) and P_(n+2) would effectively be givenweights of 22 and also added for the P_(n) pixel position, and so on.The resulting values could then be scaled by dividing by the sum of theseventeen weights shown in FIG. 9.

[0039]FIG. 10 shows how the weighting tends to converge to a Gaussiandistribution by additional passages through box filters. E.g., a fourthbox filter (or even more box filters) could be used if desired.

[0040] Returning to FIG. 3, in video effects subsystem 101, originalpixel values in frame buffer 10 are fed through five-element box filter114, one row at a time, with the modified values placed at correspondingpositions in frame buffer 112. The values in frame buffer 112 are thenfed through five-element box filter 116, one row at a time, with thefurther modified values placed at corresponding positions in framebuffer 110. The further modified values in frame buffer 110 are then fedthrough nine-element box filter 118, one row at a time, with the resultplaced at corresponding positions in frame buffer 112. The resultingvalues, which are horizontally blurred with near Gaussian distribution,are then similarly passed through the box filters, one column at time,to provide vertically blurring as well. At some point in the processing,the values are divided by the sum of weights at divider 119 (see FIG. 9)to provide scaling.

[0041] Referring to FIG. 11, alternative video effects subsystem 140 hasa single frame buffer 142 and two five-element box filters 144, 146, anine-element box filter 148, and a divider 150 all in sequence toachieve the passes through multiple box filters without intermediatestorage in frame buffers. The result, which provides horizontalblurring, would need to be stored in frame buffer 110 (after sufficientpipelining to avoid replacing data that still needs to be accessed) sothat it could then be accessed in a column-by-column manner to providevertical blurring.

[0042] Other embodiments of the invention are within the scope of theappended claims. For example, box filters having functions other thanunity multipliers in each element can be employed, though this willresult in a loss of efficiency. Also, one could use a single box filter,or a series of two, or three (as described above) or more (e.g., four,five, or six, or even more) in series. Also, one can apply a blur to anentire-image, as described above or to a portion of the image, by onlyaccessing the pixel values for the pixels being blurred and some nearbypositions.

What is claimed is:
 1. A method of creating a blur in a digital imagebased upon a frame containing original pixel values at rows and columnsof pixel positions in said image, each said pixel position having nearbypixel positions that are near to it, said method comprising receivingoriginal pixel values in said frame of image data, and for each saidpixel position in said digital image being blurred, calculating ablurred pixel value by applying a box filter having a plurality ofelements with unity multiplier values to original pixel values for pixelpositions that include said pixel position and said nearby pixelpositions.
 2. The method of claim 1 wherein said box filter has m saidelements, and wherein said receiving includes receiving said originalpixel values in sequence at said box filter, and said calculatingincludes maintaining a current filter output value in a storage area andupdating said current filter output value each time that a new pixelvalue in said sequence is received at said box filter by adding said newpixel value to said current filter output and subtracting the pixelvalue that had been received m pixel positions earlier.
 3. A method ofcreating a blur in a digital image based upon a frame containingoriginal pixel values at rows and columns of pixel positions in saidimage, each said pixel position having nearby pixel positions that arenear to it, said method comprising receiving original pixel values insaid frame of image data, and for each said pixel position in saiddigital image being blurred, calculating a blurred pixel value bysequentially applying said original pixel values as inputs to a firstbox filter that has a plurality of elements and sequentially applyingthe outputs of the first box filter as the inputs to a second boxfilter, the outputs of said first box filter being based upon saidoriginal pixel values, the outputs of said second box filter being basedupon the outputs of said first box filter.
 4. The method of claim 3wherein said calculating includes sequentially applying the outputs ofthe second box filter as the inputs to a third box filter.
 5. The methodof claim 3 wherein said box filters have unity multiplier values.
 6. Themethod of claim 4 wherein said box filters have unity multiplier values.7. The method of claim 5 or 6 wherein said box filters each have arespective number of elements, each said box filter maintaining acurrent filter output value in a storage area and updating said currentfilter output value each time that a new sequential input is received atsaid box filter by adding said new sequential input to said currentfilter output value and subtracting the input that had been received mpixel positions earlier.
 8. A blur effects subsystem for creating a blurin a digital image based upon a frame containing original pixel valuesat rows and columns of pixel positions in said image, each said pixelposition having nearby pixel positions that are near to it, saidapparatus comprising a frame buffer for receiving said frame of originalpixel values, and a box filter having a plurality of elements with unitymultiplier values, said box filter being connected to receive saidoriginal pixel values, said box filter calculating a blurred pixel valuefor each said pixel position in said digital image being blurred basedupon original pixel values for pixel positions that include said pixelposition and said nearby pixel positions.
 9. The subsystem of claim 8wherein said box filter has m said elements, and is connected to receivesaid original pixel values in sequence at said box filter, said boxfilter including a storage area that maintains a current filter outputvalue, said box filter including an adder and subtracter that updatesaid current filter output value each time that a new pixel value insequence is received at said box filter by adding said new pixel valueto said current filter output and subtracting the value that had beenreceived m pixel positions earlier.
 10. A blur effects subsystem forcreating a blur in a digital image based upon a frame containingoriginal pixel values at rows and columns of pixel positions in saidimage, each said pixel position having nearby pixel positions that arenear to it, said apparatus comprising a frame buffer for receiving saidframe of original pixel values, a first box filter that has a pluralityof elements and is connected to sequentially receive said original pixelvalues, said first box filter having a sequential first box filteroutput based upon said original pixel values, and a second box filterthat has a plurality of elements and is connected to sequentiallyreceive said first box filter output, said second box filter having asequential second box filter output based upon said first box filteroutput.
 11. The subsystem of claim 10 further comprising a third boxfilter that has a plurality of elements and is connected to sequentiallyreceive said second box filter output, said third box filter having asequential third box filter output based upon said second box filteroutput.
 12. The subsystem of claim 10 wherein said box filters haveunity multiplier values.
 13. The subsystem of claim 11 wherein said boxfilters have unity multiplier values.
 14. The subsystem of claim 12 or13 wherein said box filters each have a respective number of elements,each said box filter being connected to receive its respective inputs insequence, each said box filter including a storage area that maintains acurrent filter output value, each said box filter including an adder andsubtracter that update said current filter output value each time that anew sequential input is received at said box filter by adding said newsequential input to said output and subtracting the input that had beenreceived m pixel positions earlier.